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1.  Mobility  -  Autonomy  -  Latency  Relationship 
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Problem  Statement 


•  Trade  space  study  of  Mobility  vs.  Autonomy  vs.  Latency 


What  is  the  Relation  between: 


"Design'  Variables: 

Objective: 

-  Delays 

-  Mobility 

-  Communication 

-  Cost 

-  Hardware 

-  Power 

-  Human 

-  Weight 

-  Autonomy 

-  Reliability 

•  Identify  means  of  enhancing  mobility 
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Data  Sources 


Degree  of 
Autonomy 

Teleop 

Autonomous 

Data  Source 

Kiosk 

Analytical  Simulation 

Vehicle 

Talon 

HMMWV 

Max  Speed 

5  mph 

67  mph 

Path  Length 

300  m 

500  m 

Latency 

CERDEC  can 

0  -  700  ms  ^  provide  0  -  2000  ms 

realistic  values 

Source  of 

Latency 

Composite  (mostly  camera 
sensor) 

Sensor  and  Controller 
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Teleop  Performance  of  Path  Following  @  <  5  mph 


Completion  Time  vs.  Latency  Speed  vs.  Latency 
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Full  Autonomy  Performance  of  Obstacle  Avoidance  @  45  mph 


•  Full  autonomy 

•  Obstacle  avoidance 

•  High  speed  (45  mph) 

•  Maintain  vehicle  stability 

•  Navigate  to  minimum  time 

•  Simulation  based 

•  400  runs 
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Mobility  vs.  Autonomy  vs.  Latency  Comparison  g  9 


*Normalized  Mobility  =  Minimum  Possible  Time  /  Actual  Completion  Time 
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Effect  of  Latency  and  Vehicle  Speed  on  Fully  Autonomous  Mobility 
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Symmetric  pattern  of  results  indicates  that  combined  latency  value  is  relevant  parameter 

Increased  vehicle  speed  results  in  decreased  region  of  success 
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Latency  Compensation  to  Improve  Fully  Autonomous  Mobility  H  0  bqeco^ 


•  Latency  can  be  compensated  for 
within  the  algorithm 

•  Compensated  latent  system  recovers 
majority  of  the  performance  of  a  zero- 
latency  system 
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Effect  of  Latency  Compensation  on  Fully  Autonomous  Mobility 
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34.22  34.32  34.40  34.48  34.75  35.19  35.18  39.56  39.70  40.23  39.75 

1000 

33.79  33.87  33.88  34.06  34.18  34.17  33.82  33.94  34.05  34.06 

34.44  34.37  34.52  34.77  35.09  39.52  39.69  39.67  40.13  39.75  39.77 

1100 

33.88  34.09  34.13  34.01  34.22  34.56  33.94  34.11  34.23  34.45 

34.26  34.52  34.72  35.15  39.35  39.36  39.77  40.26  39.75  39.78  39.95 

1200 

34.12  34.11  34.19  33.84  34.37  33.86  34.10  34.24  34.37  34.38 

34.57  34.71  35.02  39.56  39.58  39.83  40.13  39.87  39.79  39.95  40.02 

1300 

34.22  34.01  34.19  34.43  33.92  34.10  34.32  34.38  34.33  34.49 

34.81  34.99  39.55  39.61  39.68  40.09  39.75  39.85  39.92  39.95  39.99 

1400 

34.20  33.83  34.41  33.92  34.18  34.23  34.28  34.38  34.60  34.81 

35.08  39.34  39.61  39.66  40.08  39.75  39.85  39.84  39.99  39.95  40.07 

1500 

34.17  34.43  33.94  33.96  34.18  34.36  34.26  34.36  34.68  39.47 

39.58  39.56  39.74  40.15  39.75  39.85  39.94  39.98  39.97  40.15  40.26 

1600 

33.84  38.80  34.14  34.27  34.31  34.45  34.44  34.78  34.95  39.41 

39.65  39.73  40.07  39.75  39.85  39.91  39.92  39.99  40.15  40.21  40.01 

1700 

33.92  34.01  34.17  34.33  34.37  34.49  34.76  39.42  39.54  39.61 

39.80  40.12  39.82  39.76  39.95  39.91  40.01  40.07  40.19  40.05  40.01 

1800 

34.00  34.19  34.39  34.25  34.59  34.82  35.07  39.35  39.64  39.69 

40.13  39.75  39.85  39.76  39.94  40.03  40.16  40.26  40.06  40.01  40.12 

1900 

34.23  34.42  34.52  34.54  34.73  35.14  35.20  39.60  39.72  40.07 

39.75  39.83  39.89  39.89  39.99  40.00  40.20  39.94  40.00  40.10  40.13 

2000 

34.32  34.40  34.60  34.76  35.09  39.33  39.66  39.76  40.04  39.81 

39.85  39.89  39.92  40.02  40.11  40.27  40.05  40.05  40.07  40.14  40.27 

15  m/s,  100m  Lidar  Range,  10m  update  spacing 


Green  =  successful  run  with  numerical  value  indicating  travel  time 

Symmetric  pattern  of  uncompensated  results  indicates  that  combined  latency  value  is  relevant  parameter 
Compensating  for  delays  expands  region  of  success 

Travel  time  for  4s  composite  compensated  delay  is  equivalent  to  ~800ms  of  uncompensated  delay 
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Trade  Space  Summary 


*  Teleop  is  inferior  to  full  autonomy 

-Teleop  completion  time  increases  with  latency 

-  Teleop  speed  decreases  with  latency 

-  Full  autonomy  speed  &  time  are  robust  against  latency 

-  Full  autonomy  can  compensate  for  latency 

-  Full  autonomy  can  work  at  high  speeds 
However,  full  autonomy  may  not  be  realizable 

•  Hence,  a  high  degree  of  (semi-)  autonomy  recommended 
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2.  Machine  -  Human  Partnership 
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Problem:  Machine  -  Human  Partnership 


■  Q1 :  Can  a  remote  human  in  conjunction  with  a  machine  beat  a 
human  or  human  team  in  their  environment? 

■  Q2:  How  do  we  identify  those  military  skills  that  are  possible? 

■  Q3:  What  feedback  (visual:  direct,  birdseye  view,  audio)  does  the 
remote  human  need  for  adequate  SA? 
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Experiment  Description 


■  Investigate  the  mobility  performance  of  a  HMMWV  driven  by 

•  an  on-board  driver 

•  a  remote  driver 

■  Drive  the  vehicle  on  smooth  and  rough  roads  and  evaluate 
limiting  performance  corresponding  to  each  driving  mode 


■  Compare  remote-driver  mobility  vs.  on-board  driver  mobility  over 
each  of  the  two  roads 
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Assumptions 


■  Vehicles  driven  by  on-board  driver  and  remote  driver  are 

identical 


■  Remote  driver  mode  has  ideal 

CERDEC  models  can  inform 

■  Sensor  suite 

changes  to  these  assumptions 

■  Perception  /  Processor  capability 

■  Communication  network 

OR 

CERDEC  research  can  provide 

■  Situational  awareness 

■  Zero  latency 

these  conditions  in  the  future 

■  Wide  bandwidth 

or  under  certain  scenarios? 

■  Benefit 

■  Vehicle  design  elements  in  manned  vehicles  can  be  modified  or 
eiiminated  in  unmanned  vehicles 
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Mobility  Limiting  Conditions 


■  Vehicle  limiting  conditions  applicable  to  both  driving  modes 

■  Engine  limit 

■  Brake  limit 

■  Tire  limit 

■  Stability  limit 

■  Structural  durability  limit 

■  Driver  limiting  condition  applicable  to  on-board  driver  mode  only 

■  Human  vibration  limit 
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Speed  Profiles  of  On-Board  and  Remote  Drivers  H 


■  On-board  driver  mode  is  significantly  hampered  by  vibration  limit  on  rough  road 

■  Therefore,  remote  driver  mode  performs  significantly  better  than  on-board  driver  mode 
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Acceleration  (G) 


Rough  Road  Driving:  On-Board  vs.  Remote  Driver 


On-Board  Driver  (16  mph)  Movies  Remote  Driver  (50  mph) 
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Machine  -  Human  Partnership  Summary 


■  It  is  hypothesized  that  a  remote  human  in  conjunction  with  a 
machine  can  beat  a  human  or  human  team  in  their  environment. 

■  On  smooth  roads,  remote  driver  mobility  is  shown  to  be  equal  or 
better  than  on-board  driver  mobility. 

■  On  rough  roads,  remote  driver  mode  is  shown  to  perform 
significantly  better  than  on-board  driver  mode. 

■  In  addition,  remotely  driven  vehicle  design  can  be  modified  and 
light  weighted  resulting  in  further  performance  enhancement. 

■  These  results  have  been  derived  under  ideal  remote  operating 
conditions  such  as  adequate  situational  awareness,  latency-free 
communication  network,  and  others  as  listed. 
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3.  Development  of  Shared  Control  Simulation  Capabiiity 
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Semi-Autonomous  UGV  Simulation 


•  Proof-of-concept  software  being  created  at  TARDEC 

-  Intent  is  to  test  feasibility  of  computer  simulation  of  all  components  of  a 
semi-autonomous  UGV 

-  Simulation  Components:  human  operator  model,  autonomous  control 
algorithm,  shared-control  algorithm,  vehicle  and  sensor  modeling 

-  Components  being  integrated  obtained  from  currently  and  previously 
funded  work  as  well  as  open  source  software  /  models 

•  JPL  ROAMS  being  modified  to  TARDEC  specifications  as  potential 
production  software  for  high-fidelity  semi-autonomous  system 
research 

-  Incorporating  new  terramechanics  models,  human  operator  modeling, 
autonomous  control,  and  shared-control  algorithms  tested  in  the  TARDEC 
proof-of-concept  software 
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Proof-of-Concept  Semi-Autonomous  Simulation 


Sensor  Data  Models 


Teleop 
Comm. 
RX  Delay 


Map  of  regions, 
surface  properties 


Terrain  and 

obstacle 

identification 


Sensor  Processing 


■>)) 


Autonomous 
Navigation  Algorithm 

UMich 


Human 
Cognitive 
Model 


Environment  Sensing 
(Lidar,  Radar,  Camera,  etc) 


Autonomous 

command 

- ► 


Teleop  Comm.  TX  Delay 


TARDEC 


Driver 

command 


Shared 

Shared 

Control 

command 

UGV  Model 

Algorithm 

TARDEC/MIT 

Vehicie  state 


Items  in  red  are  areas  where  CERDEC  can  inform  TARDEC  simulations 
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NASA/JPL  ROAMS  Capabilities  Summary 


Vehicle  Platforms:  Single  and  multi-vehicle 
simulations;  parameterized  model  templates 


Motion:  Vehicle  mobility,  arm  models,  wheel/soil 
dynamics  -  slippage/sinkage 


Hardware  models:  Kinematics,  dynamics, 
motors,  encoders,  IMU,  inertial  sensors 


Camera  sensors:  Image  synthesis  for  cameras 
with  non-idealities,  rover  and  terrain  shadows 


Environment:  SimScape  synthetic,  empirical  & 
analytic  terrains,  ephemerides  interface  for  sun 
position 


Closed-ioop  visuaiization:  Dspace  3D  graphics 
(CAD/auto-generated  vehicle  models),  data 
monitoring 


Workstation/embedded  use:  C-i-i-  &  Python 
interface  for  configuring  and  closing  the  loop  with 
software;  Stand-alone  Monte-Carlo  capability. 


Faster  than  reai-time:  6x  dynamics,  sub-second 
camera  image  synthesis 


White  and  biack  box  simulation  modes 


Hazard  ffiaps 


ATHLETE 


Descending  Victoria  Crater 


Lunar  Surface  Vehicle 


ATRV  Jr 


Lunar  Lander  &  Transporter 


Camera  &  Stereo  Models 
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Driver  Cognitive  Model  in  ACT-R 


ACT-R  Cognitive  Architecture 

•  High-level  computational  model  of  human  cognition 

•  Developed  at  Carnegie  Mellon  University  by  group 
led  by  John  Anderson 

•  Almost  40  years  of  continuous  development 
•Validated  and  updated  based  on  human 
experimentation,  brain  imaging,  and  other  studies 

•  Broad  base  of  users 

•  US  Govt  users  include:  AFRL,  ARL,  NASA  Ames,  NRL, 
NUWC,  NIST,  ONR,  Sandia  Natl  Lab 


Driver  Model 

•  Existing  driver  model  being  leveraged  for  current  effort 

•  Developed  by  Salvucci,  et  al.  at  Drexel  University  (http://cog.cs.drexei.edu/act-r/index.htmi) 

•  Models  sensory/motor  performance  of  human  driver  or  teleoperator 
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Model  Details 


•  Driver  model  incorporates  processing  of  changing  visual  locations 
with  empirically  derived  vehicle  control  laws 

-  Visual  processing  controlled  by  base  ACT-R  cognitive  model  parameters 

-  Vehicle  control  laws  part  of  Highway  Driving  Task  Model 


Visual  field  processed  to  extract 


Visual  Input 

(camera  feed  for  teleop) 


•  Teleop  Camera  Processing  Delay  Information  extracted  from 

•  Teleop  Camera  Link  Quality,  Bandwidth  visual  field  by  brain 

•  Teleop  Comm.  RX  Delay 


Control  Output 

(steering  and  throttle) 

t 

•  Teleop  Comm.  TX  Delay 
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Cognitive  Model  Path  Following  with  Latency 


*  Cognitive  model  can  incorporate  latency 

*  Can  incorporate  camera  link  quality  in  future 

*  Latency  effects  on  cognitive  model  performance  mirror  human  test  results 


Vehicle  speed:  20  m/s 

Assume  ideal  teleop  camera  quality 

Black  = 

goal  path 

Blue  = 

Cog.  w/o  delay 

Green  = 

Cog.  w/  250  ms  delay 

Red  = 

Cog.  w/  500  ms  delay 

20 


15 


-15 


.20 1 - ^ ^ ^ ^ ^ ^ ^ ^ ^ - 

0  50  100  150  200  250  300  350  400  450  500 

X  coordinate,  m 


Deviations  from  goal  path  increase  non  linearly  with  delay 
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UMich  (ARC)  Autonomy  Algorithm  Overview 

Possible  CERDEC  Models  (Data  Quality,  Processing  Delays) 


•  Auxiliary  Data 

•  Bounds 

•  Initial  Mesh 
Obstacle  •  Initial  Solution  Guess 

Information 


Control  Problem 


Problem  I 
Specifications 


Optimal 

Control 

Problem 


Optimal 

Control 

Commands 


Control  Problem 
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y<m^ 


Effect  of  Sensor  Modeling  on  Autonomy 


143  15D  16D 


31  p] 


NATO  Double  Lane  Change  Scenario 


Longitudinal  speed: 
Prediction  horizon: 
Execution  horizon: 


20  m/s 
2.5  seconds 
0.2  second 


2D  LIDAR  Model 

Algorithm  fails 


Pseudo-3D  LIDAR 

(assumes  full  knowledge  out 

to  fixed  range) 

Algorithm  succeeds 


Realistic  3D  Lidar 
Expected  performance 
falls  between  two 
extremes. 


Algorithm  result  =  ? 


130 


120 


x(m) 


*  Accurate  algorithm  assessment  and  optimization  require  good  sensor  models 
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Mobility  Scenarios  and  Metrics 


NATO  Double  Lane  Change 


•  •  • 


L 


Urban  Navigation 


Off-Road  Mobility 


*  Mobility  metrics 

-  Minimum  time 

-  Maximum  speed 

-  Go/NoGo 

*  Failure  modes 

-  Rollover 

-  Immobilization 

-  Collision 
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Intelligent  Vehicle  Mobility  Simulation  Roadmap  H  0 
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4.  Conclusion 
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CERDEC  Expertise 


Communication  Network 
Modeling 


For  use  in  cognitive  modeling  and  potential 
mitigation  techniques  such  as  predictive 
displays 

Communication  link  parameters 

Latency  distribution 
Bandwidth  (i.e.  video  quality) 
Incorporate  scenario  (urban,  off-road) 


Sensor  Modeling 


For  use  in  cognitive  modeling  and  potential 
mitigation  techniques  such  as  predictive 
displays 

Sensor  Data  Parameters 

Camera  model 
IMU/GPS  models 
Sensor  Processing  Delays 


For  use  if  autonomous  algorithm  is  using  cloud- 

based  computation  or  needs  to  be  sent  changed  To  accurately  model  inputs  to  autonomous 
mission  goals  algorithm  and  allow  for  possible  mitigation 


Communication  link  parameters 

Latency  distribution 
Bandwidth  (i.e.  max  data  rate) 
Incorporate  scenario  (urban,  off-road) 


Sensor  Data  Parameters 
LIDAR  /  Camera  /  Radar  models 
IMU/GPS  models 
Sensor  Processing  Delays 


UNCLASSIFIED:  Distribution  Statement  A.  Approved  for  public  release.  #26550 


36 


Possible  Next  Steps 


Near  Term 

•  CERDEC  runs  connnnunication  network  models  for  current  mobility 
scenarios  of  interest  and  provides  latency  statistics  for  incorporation  into 
TARDEC  models 

-  LOS  and  beyond-LOS  links  for  teleoperation  in  urban  and  off-road 
environments;  single  operator  /  single  vehicle 

-  Latency  statistics 

•  Determine  what/whether  appropriate  CERDEC  sensor  models  can  be 
provided  in  a  form  usable  by  current  TARDEC  simulation  software 

Longer  Term 

•  Investigate  possibility  for  co-simulation  between  CERDEC  and  TARDEC 
software  for  mobility  simulation 

•  Develop  complex  scenarios  and  vehicle  teaming  for  mobility  simulation 
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